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THE EFFECT OF ADDITIONAL OBSERVATIONS 
ON A PREVIOUS LEAST SQUARES ESTIMATE 


INTRODUCTION 

In a number of applications of the method of least squares it may- 
be desirable to compute the influence of additional observations on a 
previously determined least squares estimate of the regression para- 
meters and their associated covariance matrix. A particular applica- 
tion is presented in which the parameters are assumed to be constants, 
although the method can be extended to dynamic systems as shown by 
Gainer in reference [1] . Matrix equations which provide for additional 
observations are derived for the method of weighted least squares with 
uncorrelated errors. These recursion formulas reduce to the ones 
presented by Gainer in reference [1] in the case where only a single 
observation is added. 

DERIVATION OF METHOD 

The nonlinear estimation problem is usually solved by assuming a 
linear approximation of the true regression parameters in a neighbor- 
hood of a nominal or reference set of parameters. Let us assume that 
we have obtained a least squares estimate and also the covariance matrix 
of the regression parameters. ^We are concerned here with the problem 
of determining a revised least squares estimate and its associated co- 
variance matrix when additional observations become availableTj 

Consider the following system of n + p observation equations re- 
lating deviations in "weighted'' observations to deviations in the k 
(n > k) unknowns 
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Now let us apply the method of partitioning to the matrix equation 
AX = Y + e, (2) 
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and define 



f A 


y 

n J n 


+1 y n + 1 

W + p y n + P / 




( 3 ) 


where 

A n is a n x k matrix of coefficients a { . , (i = 1, . . n; j = 1, 

• • • , k), 


B p is a p x k matrix of coefficients a.. , (i = n+ 1, . . n + p; 
j - 1 k), 

E is a n x k matrix of coefficients /w. a . . , (i = 1, . . .. n; 

n l l j v 

j - 1. • • • . k), 

F p is a p x k matrix of coefficients VW A a i . , (i = n + 1, . . 
n + p; j = 1 k), 

u n is a n x 1 vector of weighted observations, 

v p is a p x 1 vector of additional weighted observations, 

is a n x 1 vector of errors in weighted observations, 

£> 2 is a p x 1 vector of errors in additional weighted observations, 

W n is a n x 1 diagonal "weighting " matrix. 
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Let us assume that the errors in the observations are random vari- 
ables having a Gaussian distribution with zero mean and E [ee T ] = Q, 
where Q is a diagonal covariance matrix with elements cr. 2 . It can be 
shown that the least squares estimate of X which depends upon n ob- 
servations is given by 


X, = (E„ T E„) 
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where the diagonal "weighting" matrix is given by 



Now C„ = e[(x„ - X) (x„ - X)T] = (A„T» n A n )-' = (EJEj-* 

is the covariance matrix of X n , and it is important for its statistical 
information. Let X. , i = 1, 2, . . . , k, be the parameters to be 

estimated. Then an unbiased estimate of the variance of X £ is given by 



e T W e 
N - k b “ 


(5) 


where 

b u is the ith diagonal element of C n , 

N is the number of observation equations, 

k is the number of parameters to be estimated. 

It would be of interest to obtain a revised estimate of X n and C n 
by considering p additional observations. The estimate of X which 
depends upon n + p observations- is 



Since C ** - (E T E ) we write 

n v n n ' k x k 

(®„ T *„ + V F ,)-‘ = fc; 1 + V f p)'' = d-«)-‘c„ (8) 

where H = - C n F p T F p . 

Substitution of (8) into (7) gives 

= (i-»r‘c(E> n( F;vj (9) 

= (I - H)‘* (C n E n T u n + C n F p T v p ) 

= (I - »)- k ‘, k (X n +C n F p T Tp ) t , 1 . (10) 

Evaluation of (I - H)' 1 

For the inversion of the matrix (I - H) we find application of the 
following formula in reference [2] 
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(a + U S - 1 V 1 )- 1 


A ' 1 - A ' 1 U (S + V T A * 1 u )* 1 VTA ' 1 


( 11 ) 


where A and S are nonsingular matrices, U and V are rectangular. 
Putting 


A = I„. k . s = 


I U = C F T V T 

p X p > u '"'n p ’ V 

k x p k*k k x p 
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the form of (11) becomes 


(I - HV 1 , = I - C F T ( I + F C F 1 )' 1 

7 k x k 1v . n PV p n p ) 


F (12) 

k x k r r r ' p 

kxk k x p p x p p x k k x k kx pp x p p xk 


Substitution of (12) into (10) gives 


x n . = [i - C F D T (I + F C F t ) X F 1 i + c F T V 

n+ P L n p \ P n p / pj n npp 


(13) 


We may select either of the equations (10) or (13) for the evaluation 
of X n + . It is to be observed that in equation (10) one always inverts 

a matrix of order k, where k is the number of unknowns; whereas in 
equation (13) it is necessary to invert a matrix of order p, where p is 
the number of additional observations. Consequently, if the situation 
arises where k is much greater than p, it would probably be desirable 
to use equation (13). 

Special Case When p = 1 


If we select p = 1 we find that (13) reduces to 


X +1 = X + 

n + 1 n 


C F 1 

ii r i 


1 + F, C F . 1 

1 n 1 
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(14) 
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where (f i C n Fj T ) is now a scalar quantity for the case of a single addi- 
tional observation. We see that (14) is equivalent to equation (36) in 
reference [1] . We can also observe that the estimate X n + t does 

not require any matrix inversion. Hence it is clear that the recursive 
equation (14) may be used as a least square sequential estimation 
procedure to obtain a revised estimate for the parameter vector. 


Evaluation of C , 

n + p 

The revised covariance matrix C n + is given by 
C + = (E T E + F t F)' 1 = (I - H)* 1 C 

n+p \nn p p / v / n 

= [i - C F T (I + F C F T r X F 1 C . 

L n p \ P n p / pjn 

Special Case When p = 1 

Let p = 1. Then we find that (15) reduces to 

C F. T F. C 

n 1 In 

c = c - • 

n + 1 n 1 + F. C F. t 

1 n 1 


(15) 


( 16 ) 


Let J 

k x 1 
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k x k 


F i T 

k x 1 


Then (16) becomes 


C 


n + 1 


J J T 

1 + F i J 


(17) 


It is to be observed that ( 1 6 ) is equivalent to equation (34) in refer- 
ence [1] and that it does not require matrix inversion. 


6 



GEOMETRICAL INTERPRETATION OF THE LEAST 
SQUARES PROBLEM 

The presentation which follows utilizes certain basic concepts set 
forth by Householder in references [2] and [3] in providing a geometric 
interpretation of the method of weighted least squares. 

Consider the linear least squares problem where one has given a 
n x k matrix A with n > k rows and a n- vector y. One desires a 
k- vector X such that 

AX = y+e, P T e = 0 


where 

e is the vector of residuals, 



Let us now consider the " weighting" matrix W as a metric for the 
space and hence define the orthogonality and lengths of the vectors with 
respect to this metric. One can give a geometrical interpretation to 
the method of least squares, i. e. , the projection of an arbitrary given 
vector upon a certain subspace. Let the columns of the rectangular 
matrix A be linearly independent. Then the vector AX is a vector in 
the same space spanned by the columns of A and AX is said to be the 
orthogonal projection with respect to W of the vector y whenever y - AX 
is orthogonal to A with respect to the positive definite and symmetric 
matrix W, i. e. , 

A T W (y - AX) = 0 

or X = (A T W A)* 1 A t W y, 

which is exactly the least squares solution of the usual normal equations. 
The assumption that the columns of A are linearly independent together 
with the fact that W is a positive definite symmetric matrix leads to the 
following definition of a generalized projector reference [3] . 

Definition: A matrix T A is the generalized projector with respect to W 
for the space of A if T A . ff = A (A T W A)' 1 A t W. 
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The generalized projector has the following properties: 

(a) T A;W is a unique projector since if A i s replaced by AK, 

where K is any nonsingular matrix, one obtains the same 
projector. This can be verified directly, i. e. , 

T AK; w = AK [< AK ) T W (AK )]' 1 ( A K) T W 

= AK [K T ( A T WA) k ]* 1 K t A t W 
= A (KK- 1 ) (A t WA)* x (k t )* X K t A t W 
T AK;W = A ( A T WA) ’ X ATW = T A;W 

(b) WT A . w is symmetric, i. e. , (WT A . w ) T = WT A;W 

Now (WT a . w ) t = [WA (A T WA)" X A t w] t = WA (A t WA)‘ x A t W 
= WT 

W X A; W 

2 

(c) T A;W is idempotent, i. e. , T A;W = T A;W . 

T a . w = A (A T WA)' X ATW A (A t WA)* X A t W = A (A T WA)* X A t W 

= T 

A A; W 


(d) T A . w y is the orthogonal projection of the vector y . This is 
shown by taking the generalized scalar product of T A . w y and 
the residual vector (y - T A;W y) , i. e. , 

(T A; „y) T * (I - T #i „) y = y T (wt A; „) t (I - T Al „) y 

= y T »T J;> (I - T 4i ,)y = y T " ( T *.» ■ T »i« ) y - o 
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since by (b) WT A;W is symmetric and by (c) T A;W is 
idempotent. 
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